Modeling Stock Order Flows and Learning Market-Making from Data

Stock markets employ specialized traders, market-makers, designed to provide liquidity and volume to the market by constantly supplying both supply and demand. In this paper, we demonstrate a novel method for modeling the market as a dynamic system and a reinforcement learning algorithm that learns profitable market-making strategies when run on this model. The sequence of buys and sells for a particular stock, the order flow, we model as an Input-Output Hidden Markov Model fit to historical data. When combined with the dynamics of the order book, this creates a highly non-linear and difficult dynamic system. Our reinforcement learning algorithm, based on likelihood ratios, is run on this partially-observable environment. We demonstrate learning results for two separate real stocks. Copyright c Massachusetts Institute of Technology, 2002 This report describes research done at the Center for Biological & Computational Learning, which is affiliated with the McGovern Institute of Brain Research and with the Artificial Intelligence Laboratory, and which is in the Department of Brain & Cognitive Sciences at MIT. This research was sponsored by grants from: Office of Naval Research (DARPA) under contract No. N00014-00-1-0907, National Science Foundation (ITR) under contract No. IIS-0085836, National Science Foundation (KDI) under contract No. DMS-9872936, and National Science Foundation under contract No. IIS-9800032. Additional support was provided by: AT&T, Central Research Institute of Electric Power Industry, Center for e-Business (MIT), DaimlerChrysler AG, Compaq/Digital Equipment Corporation, Eastman Kodak Company, Honda R&D Co., Ltd., ITRI, Komatsu Ltd., Merrill-Lynch, Mitsubishi Corporation, NEC Fund, Nippon Telegraph & Telephone, Oxygen, Siemens Corporate Research, Inc., Sumitomo Metal Industries, Toyota Motor Corporation, WatchVision Co., Ltd., and The Whitaker Foundation. C.S. was supported by ONR contract N00014-00-1-0637 under the MURI program “Decision Making under Uncertainty” while at Stanford. Introduction Many economic markets, including most major stock exchanges, employ market-makers to aid in the transactions and provide a better quality market. Each commodity has one or more market-makers assigned to it. The marketmaker’s responsibility is to set prices and volumes for buying and selling. In particular, a market-maker will constantly quote a bid price and an ask price (the prices at which the market-maker is willing to buy or sell respectively) as well as associated sizes (the maximum volume to which the market-maker is committed at that price). Market-makers supply an advantage to the market. By consolidating the trading in a few agents, the market becomes more efficient. Traders wishing to buy or sell do not need to find each other or wait for each other’s arrival. Additionally, by quoting a single price which is guaranteed for all traders, market-makers can help to smooth price fluctuations due to spurious supplies or demands. Market-makers benefit themselves from their position. They have an informational advantage over because they see more of the orders than an average trader. This is offset either by institutional regulations or by competition from other market-makers. Although they serve as a clearing house for orders, they make trades against their personal cash and inventory to cover their own quotes. Such a position carries risk. Many major markets are now electronic. The NASDAQ is a distributed trading system completely run through networked computers. It uses competing market-makers to maintain a high quality market. However, the demands on human market-makers are high. A typical market-maker is responsible for 10 to 20 securities. At any given moment, it is only feasible for the human to be actively attentive to 2 to 3 of them. The market-maker is generally losing potential profit or volume on the other securities. Many other smaller markets (on-line and off-hour trading systems) have emerged as a result of the recent increases in computer networking. These systems are usually too small to employ human market-makers. Instead, orders are crossed against other orders that happen to be present at the time of the trade. This results in many unfilled orders and the loss of potential customers. The goal of this paper is to explore the potential for automated electronic market-making. For the case of established systems like the NASDAQ, such a system could fill the role of an “autopilot” by taking care of stocks in an intelligent manner while being supervised by a human. This would allow a single human to more successfully manage a large set of securities. For the case of small on-line trading systems, such an automated system could replace the existing naive order crossing mechanism to provide a better market to its traders. Previous Work There has been much work on understand both the price formation process and market-making strategies. There are two main approaches from the economics literature. One focuses on the uncertainties in the order flow (the time series of orders placed) and the inventory of the market-maker (Garman 1976; Amihud & Mendelson 1980; Ho & Stoll 1981; O’Hara & Oldfield 1986). The other tries to explain the the price setting dynamics using information-based models (Glosten & Milgrom 1985). Most of these studies have developed conditions for optimally but provided no explicit price adjustment policies. For example, in Amihud & Mendelson (1980), bid and ask prices are shown to relate to inventory, but the exact dependence is unavailable. Some analyses do provide functional forms for marketmaking policies (O’Hara & Oldfield 1986), but the practical application of the results is limited due to stringent assumptions made in the models. In previous work (Chan 2001; Chan & Shelton 2001; Shelton 2001a), we developed a simple information-based market model (similar to that of Glosten & Milgrom) and employed reinforcement learning to derive explicit marketmaking policies. The reinforcement learning algorithm knew nothing of the underlying assumed model and therefore its application was general. By using a flexible method based on experience, we hoped that we could apply the same algorithm to more complex models with differing dynamics without change. Contribution In this paper, we continue this research by building a more complex market model based on real market data. We employ the same reinforcement learning algorithm that we used in previous information-based models to derive a profitable market-making strategy. This strengthens our statement that by using a learning strategy we are assuring that the algorithm’s success is not predicated on the exact details of our market model. Modeling the Stock Order Flow In order to argue how well the market-making model will perform in the real market, it is important to test under a realistic market environment. Previously, we tested under a simple environment where number of unrealistic assumptions were involved. Although the algorithm performed well in such market, the market assumption (i.e. the existence of a true price process and differentiation of the informed and uninformed traders) over-simplified the model. Additionally, the trading crowd did not react to any of the market-maker’s actions nor their previous actions. In this paper, we propose a new approach to empirically model the aggregated behavior of the trading crowd. Practically, this involves modeling the time sequence of orders placed on the market, called the order flow. Our new approach is more realistic than the previous model for two reasons. First, the new model does not assume any “true” pricing process. In the real market, the true price of a stock is generally unknown; the price emerges from the aggregated behaviors of the trading crowd. Each trader has a different preference toward the stock and their behavior can change based on their private or public information. Such information is disseminated or aggregated by the trading process. The trading process is the source of information to traders (Easley, Kiefer, & O’Hara 1997). Second, the new model will react to the market-maker’s quotes. The quoted bid and ask prices and the corresponding bid/ask spread (the difference between the two values) are relevant to trading crowd’s decision making process. Therefore, in this paper we will generate orders based on probability distribution conditioned by the trading process and the market-maker’s quotes. TORQ Dataset We fit our model to the transaction-level historical dataset Trade, Order, and Quote (TORQ) database from New York Stock Exchange (NYSE). The TORQ dataset consists of trade, order, and quote information for 144 stocks traded at NYSE from November 1990 to January 1991. The trade and quote data show the transactions and changes of price and size of bid/ask in the market. Although TORQ contains 100% of trade and quote data, full order data are not available in the dataset. The order data covers only the orders that entered through the NYSE’s electronic order routing system (SuperDot system). Other orders, which are usually larger, are assisted by floor brokers. In 1992, 75% of orders entered the market via SuperDot but only accounted for 28% of the executed share volume (Hassbrouk, Sofianos, & Sosebee 1993). Due to such limitation in our dataset, several assumptions are needed to be made in our order flow model. We will discuss this more in the later section. Variables and Assumptions Orders. The order flow model generates 4 types of orders: market buy, market sell, limit buy and limit sell. To define such orders, we introduce 4 variables: Arrival Time: The time difference between current order and preceding order. Size: The number of shares of an order. Side: Defines whether an order is buy or sell. Price: The price of an order. This only applies to limit orders. We assume that the orders are stochastically generated conditioned on the market conditions. If the same market conditions are given as the training data, the generated orders will form a similar trading process. If different market conditions are given, the system will generate a sequence of orders that react to the new conditions. Market condition. The market conditions will be determined by the orders that are already generated and by the market-maker’s actions. We will introduce 5 variables, which represent the market conditions at time : Difference between bid and ask price (bid/ask spread): . Difference between bid and ask size: . Return of transaction price: ! " #!$